{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 2: Understanding and Using CycleReviewer\n",
    "\n",
    "## Introduction\n",
    "\n",
    "CycleReviewer (also known as WhizReviewer) is a set of specialized large language models that have undergone extensive supervised training specifically for academic peer review. The model is available in three sizes:\n",
    "\n",
    "- 8B parameters (based on Llama3.1)\n",
    "- 70B parameters (based on Llama3.1)\n",
    "- 123B parameters (based on Mistral-Large-2)\n",
    "\n",
    "These models are designed to simulate the peer review process by evaluating research papers across multiple dimensions, providing detailed feedback, and generating comprehensive reviews that closely mirror those produced by human academic reviewers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting Up the Environment\n",
    "\n",
    "Let's start by importing the necessary libraries and initializing the CycleReviewer model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO 02-26 00:33:07 __init__.py:183] Automatically detected platform cuda.\n",
      "INFO 02-26 00:33:09 config.py:2364] Downcasting torch.float32 to torch.float16.\n",
      "INFO 02-26 00:33:16 config.py:526] This model supports multiple tasks: {'classify', 'embed', 'score', 'reward', 'generate'}. Defaulting to 'generate'.\n",
      "WARNING 02-26 00:33:16 arg_utils.py:1119] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.\n",
      "INFO 02-26 00:33:16 config.py:1538] Chunked prefill is enabled with max_num_batched_tokens=2048.\n",
      "INFO 02-26 00:33:16 llm_engine.py:232] Initializing a V0 LLM engine (v0.7.1) with config: model='/zhuminjun/llama70/CycleReviewer-8B', speculative_config=None, tokenizer='/zhuminjun/llama70/CycleReviewer-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=50000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/zhuminjun/llama70/CycleReviewer-8B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={\"splitting_ops\":[],\"compile_sizes\":[],\"cudagraph_capture_sizes\":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],\"max_capture_size\":256}, use_cached_outputs=False, \n",
      "INFO 02-26 00:33:17 cuda.py:235] Using Flash Attention backend.\n",
      "INFO 02-26 00:33:18 model_runner.py:1111] Starting to load model /zhuminjun/llama70/CycleReviewer-8B...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8f30fffb6c6346a285ce061d7238d256",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading safetensors checkpoint shards:   0% Completed | 0/7 [00:00<?, ?it/s]\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO 02-26 00:36:30 model_runner.py:1116] Loading model weights took 14.9888 GB\n",
      "INFO 02-26 00:36:31 worker.py:266] Memory profiling takes 0.64 seconds\n",
      "INFO 02-26 00:36:31 worker.py:266] the current vLLM instance can use total_gpu_memory (79.14GiB) x gpu_memory_utilization (0.95) = 75.18GiB\n",
      "INFO 02-26 00:36:31 worker.py:266] model weights take 14.99GiB; non_torch_memory takes 0.09GiB; PyTorch activation peak memory takes 1.19GiB; the rest of the memory reserved for KV Cache is 58.91GiB.\n",
      "INFO 02-26 00:36:31 executor_base.py:108] # CUDA blocks: 30163, # CPU blocks: 2048\n",
      "INFO 02-26 00:36:31 executor_base.py:113] Maximum concurrency for 50000 tokens per request: 9.65x\n",
      "INFO 02-26 00:36:33 model_runner.py:1435] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Capturing CUDA graph shapes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:15<00:00,  2.27it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO 02-26 00:36:49 model_runner.py:1563] Graph capturing finished in 15 secs, took 0.26 GiB\n",
      "INFO 02-26 00:36:49 llm_engine.py:429] init engine (profile, create kv cache, warmup model) took 18.18 seconds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "from ai_researcher import CycleReviewer\n",
    "\n",
    "# Initialize CycleReviewer with the 8B parameter model (for faster inference)\n",
    "# You can choose \"8B\", \"70B\", or \"123B\" depending on available computational resources\n",
    "reviewer = CycleReviewer(model_size=\"8B\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading a Paper to Review\n",
    "\n",
    "Let's load a paper from a JSON file and prepare it for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paper Title: Scientific Peer Reviewer Agents}\n",
      "\n",
      "\n",
      "Abstract length: 1177 characters\n",
      "LaTeX content length: 41664 characters\n"
     ]
    }
   ],
   "source": [
    "# Load a paper from our JSON file\n",
    "with open('generated_paper.json', 'r', encoding='utf-8') as f:\n",
    "    papers = json.load(f)\n",
    "\n",
    "# Print basic information about the paper\n",
    "print(f\"Paper Title: {papers[0]['title']}\")\n",
    "print(f\"Abstract length: {len(papers[0]['abstract'])} characters\")\n",
    "print(f\"LaTeX content length: {len(papers[0]['latex'])} characters\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reviewing a Single Paper\n",
    "\n",
    "Now, let's use CycleReviewer to evaluate the paper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Processed prompts: 100%|███████████████████████████████████████████████████████████████████████| 10/10 [02:44<00:00, 16.45s/it, est. speed input: 571.37 toks/s, output: 316.69 toks/s]\n"
     ]
    }
   ],
   "source": [
    "# Generate the review\n",
    "review_result = reviewer.evaluate([paper['latex'] for paper in papers])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing and Analyzing Review Results\n",
    "\n",
    "Let's parse the review to extract structured information including ratings, strengths, weaknesses, and recommendations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: Scientific Peer Reviewer Agents}\n",
      "\n",
      "\n",
      "Average Rating: 4.00/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 4\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 1.0\n",
      "  Reviewer 2: 5.0\n",
      "  Reviewer 3: 5.0\n",
      "  Reviewer 4: 5.0\n",
      "---***---\n",
      "Title: Multi-Agent Review: Simulating Human Reviewers for Scientific Peer Review with Large Language Models}\n",
      "\n",
      "\n",
      "Average Rating: 3.00/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 4\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 3.0\n",
      "  Reviewer 2: 3.0\n",
      "  Reviewer 3: 3.0\n",
      "  Reviewer 4: 3.0\n",
      "---***---\n",
      "Title: Referees in AI for Scientific Peer Review}\n",
      "\n",
      "\n",
      "Average Rating: 4.00/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 4\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 1.0\n",
      "  Reviewer 2: 5.0\n",
      "  Reviewer 3: 5.0\n",
      "  Reviewer 4: 5.0\n",
      "---***---\n",
      "Title: Evaluating LLM-based AI Reviewer Agent for Scientific Peer Review}\n",
      "\n",
      "\n",
      "Average Rating: 5.00/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 4\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 5.0\n",
      "  Reviewer 2: 5.0\n",
      "  Reviewer 3: 5.0\n",
      "  Reviewer 4: 5.0\n",
      "---***---\n",
      "Title: AI-Powered Peer Review Can Help Scientific Progress}\n",
      "\n",
      "\n",
      "Average Rating: 4.50/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 4\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 3.0\n",
      "  Reviewer 2: 5.0\n",
      "  Reviewer 3: 5.0\n",
      "  Reviewer 4: 5.0\n",
      "---***---\n",
      "Title: Optimizing AI Agents for Simulated Peer Review}\n",
      "\n",
      "\n",
      "Average Rating: 5.00/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 4\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 5.0\n",
      "  Reviewer 2: 5.0\n",
      "  Reviewer 3: 5.0\n",
      "  Reviewer 4: 5.0\n",
      "---***---\n",
      "Title: Scientific Review Protocol as Multi-agent Competition Game}\n",
      "\n",
      "\n",
      "Average Rating: 4.50/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 4\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 3.0\n",
      "  Reviewer 2: 5.0\n",
      "  Reviewer 3: 5.0\n",
      "  Reviewer 4: 5.0\n",
      "---***---\n",
      "Title: AI Researchers: AI-Powered Agents for Scientific Peer Review and Idea Generation}\n",
      "\n",
      "\n",
      "Average Rating: 3.00/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 4\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 3.0\n",
      "  Reviewer 2: 3.0\n",
      "  Reviewer 3: 3.0\n",
      "  Reviewer 4: 3.0\n",
      "---***---\n",
      "Title: Enhancing the Quality of LLM-Based Scientific Peer Review Through Interactive Learning}\n",
      "\n",
      "\n",
      "Average Rating: 3.00/10\n",
      "Decision: Reject\n",
      "Number of Reviewers: 5\n",
      "\n",
      "Individual Ratings:\n",
      "  Reviewer 1: 3.0\n",
      "  Reviewer 2: 3.0\n",
      "  Reviewer 3: 3.0\n",
      "  Reviewer 4: 3.0\n",
      "  Reviewer 5: 3.0\n",
      "---***---\n"
     ]
    }
   ],
   "source": [
    "# Parse the review\n",
    "review_data = review_result[0]\n",
    "\n",
    "for i in range(len(review_result)):\n",
    "    if review_result[i] != None:\n",
    "        review_data = review_result[i]\n",
    "        # Print summary of the review\n",
    "        print(f\"Title: {papers[i]['title']}\")\n",
    "        print(f\"Average Rating: {review_data['avg_rating']:.2f}/10\")\n",
    "        print(f\"Decision: {review_data['paper_decision']}\")\n",
    "        print(f\"Number of Reviewers: {len(review_data['reviews'])}\")\n",
    "        print(\"\\nIndividual Ratings:\")\n",
    "        for i, rating in enumerate(review_data['rating']):\n",
    "            print(f\"  Reviewer {i+1}: {rating}\")\n",
    "        print('---***---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's examine some key feedback from the reviewers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KEY STRENGTHS IDENTIFIED:\n",
      "--------------------------------------------------\n",
      "Reviewer 1:\n",
      "- The paper proposes a computational model for scientific discovery, emphasizing the critical role of scientific peer review, and introduces the Scientific Peer Reviewer Agent (SPRA) for simulating the peer review process of scientific papers. The Human-LLM Reviewer Agent (HLL-RA) is developed for iterative review optimization.\n",
      "- The impact of HLL-RA is validated through the creation of a new synthetic dataset (Syn-RS) of peer reviews for scientific papers, which are rated by humans.\n",
      "- Based on ...\n",
      "\n",
      "Reviewer 2:\n",
      "- The paper proposes a computational model for scientific discovery, emphasizing the critical role of scientific peer review, and introduces the Scientific Peer Reviewer Agent (SPRA) for simulating the peer review process of scientific papers. The Human-LLM Reviewer Agent (HLL-RA) is developed for iterative review optimization.\n",
      "- The impact of HLL-RA is validated through the creation of a new synthetic dataset (Syn-RS) of peer reviews for scientific papers, which are rated by humans.\n",
      "- Based on ...\n",
      "\n",
      "Reviewer 3:\n",
      "- The paper proposes a computational model for scientific discovery, emphasizing the critical role of scientific peer review, and introduces the Scientific Peer Reviewer Agent (SPRA) for simulating the peer review process of scientific papers. The Human-LLM Reviewer Agent (HLL-RA) is developed for iterative review optimization.\n",
      "- The impact of HLL-RA is validated through the creation of a new synthetic dataset (Syn-RS) of peer reviews for scientific papers, which are rated by humans.\n",
      "- Based on ...\n",
      "\n",
      "Reviewer 4:\n",
      "- The paper proposes a computational model for scientific discovery, emphasizing the critical role of scientific peer review, and introduces the Scientific Peer Reviewer Agent (SPRA) for simulating the peer review process of scientific papers. The Human-LLM Reviewer Agent (HLL-RA) is developed for iterative review optimization.\n",
      "- The impact of HLL-RA is validated through the creation of a new synthetic dataset (Syn-RS) of peer reviews for scientific papers, which are rated by humans.\n",
      "- Based on ...\n",
      "\n",
      "Reviewer 5:\n",
      "- The paper proposes a computational model for scientific discovery, emphasizing the critical role of scientific peer review, and introduces the Scientific Peer Reviewer Agent (SPRA) for simulating the peer review process of scientific papers. The Human-LLM Reviewer Agent (HLL-RA) is developed for iterative review optimization.\n",
      "- The impact of HLL-RA is validated through the creation of a new synthetic dataset (Syn-RS) of peer reviews for scientific papers, which are rated by humans.\n",
      "- Based on ...\n",
      "\n",
      "\n",
      "KEY WEAKNESSES IDENTIFIED:\n",
      "--------------------------------------------------\n",
      "Reviewer 1:\n",
      "- The paper is not well written. It is not clear what the main contribution of this paper is. \n",
      "- The paper does not provide a clear description of the method. \n",
      "- The paper does not provide a clear description of the experiments. \n",
      "- The paper does not provide a clear description of the results. \n",
      "- The paper does not provide a clear description of the limitations of the proposed method. \n",
      "- The paper does not provide a clear description of the future work.\n",
      "\n",
      "...\n",
      "\n",
      "Reviewer 2:\n",
      "- The paper is not well written. It is not clear what the main contribution of this paper is. \n",
      "- The paper does not provide a clear description of the method. \n",
      "- The paper does not provide a clear description of the experiments. \n",
      "- The paper does not provide a clear description of the results. \n",
      "- The paper does not provide a clear description of the limitations of the proposed method. \n",
      "- The paper does not provide a clear description of the future work.\n",
      "\n",
      "...\n",
      "\n",
      "Reviewer 3:\n",
      "- The paper is not well written. It is not clear what the main contribution of this paper is. \n",
      "- The paper does not provide a clear description of the method. \n",
      "- The paper does not provide a clear description of the experiments. \n",
      "- The paper does not provide a clear description of the results. \n",
      "- The paper does not provide a clear description of the limitations of the proposed method. \n",
      "- The paper does not provide a clear description of the future work.\n",
      "\n",
      "...\n",
      "\n",
      "Reviewer 4:\n",
      "- The paper is not well written. It is not clear what the main contribution of this paper is. \n",
      "- The paper does not provide a clear description of the method. \n",
      "- The paper does not provide a clear description of the experiments. \n",
      "- The paper does not provide a clear description of the results. \n",
      "- The paper does not provide a clear description of the limitations of the proposed method. \n",
      "- The paper does not provide a clear description of the future work.\n",
      "\n",
      "...\n",
      "\n",
      "Reviewer 5:\n",
      "- The paper is not well written. It is not clear what the main contribution of this paper is. \n",
      "- The paper does not provide a clear description of the method. \n",
      "- The paper does not provide a clear description of the experiments. \n",
      "- The paper does not provide a clear description of the results. \n",
      "- The paper does not provide a clear description of the limitations of the proposed method. \n",
      "- The paper does not provide a clear description of the future work.\n",
      "\n",
      "...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Display key strengths mentioned by reviewers\n",
    "print(\"KEY STRENGTHS IDENTIFIED:\")\n",
    "print(\"-\" * 50)\n",
    "for i, strength in enumerate(review_data['strength']):\n",
    "    print(f\"Reviewer {i+1}:\")\n",
    "    # Print the first 200 characters of each strength for brevity\n",
    "    print(strength[:500] + \"...\" if len(strength) > 200 else strength)\n",
    "    print()\n",
    "\n",
    "# Display key weaknesses mentioned by reviewers\n",
    "print(\"\\nKEY WEAKNESSES IDENTIFIED:\")\n",
    "print(\"-\" * 50)\n",
    "for i, weakness in enumerate(review_data['weaknesses']):\n",
    "    print(f\"Reviewer {i+1}:\")\n",
    "    # Print the first 200 characters of each weakness for brevity\n",
    "    print(weakness[:500] + \"...\" if len(weakness) > 200 else weakness)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Meta Review\n",
    "\n",
    "Let's look at the meta review, which synthesizes the individual reviewer feedback:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "META REVIEW:\n",
      "==================================================\n",
      "The paper proposes a computational model for scientific discovery, emphasizing the critical role of scientific peer review, and introduces the Scientific Peer Reviewer Agent (SPRA) for simulating the peer review process of scientific papers. The Human-LLM Reviewer Agent (HLL-RA) is developed for iterative review optimization. The impact of HLL-RA is validated through the creation of a new synthetic dataset (Syn-RS) of peer reviews for scientific papers, which are rated by humans. Based on not only the simulated reviews but also the reviews selected by HLL-RA, the LLMs are further fine-tuned to better align with human judgment, validated through achieving new state-of-the-art (SOTA) performance on the RewardBench benchmark.\n",
      "\n",
      "The reviewers are unanimous in their opinion that the paper is not well written, and that the main contribution of this paper is not clear. The paper does not provide a clear description of the method, experiments, results, limitations, and future work. Therefore, the paper is not ready for publication at the current stage.\n",
      "\n",
      "### justification_for_why_not_higher_score\n",
      "\n",
      "The reviewers are unanimous in their opinion that the paper is not well written, and that the main contribution of this paper is not clear. The paper does not provide a clear description of the method, experiments, results, limitations, and future work. Therefore, the paper is not ready for publication at the current stage.\n",
      "\n",
      "### justification_for_why_not_lower_score\n",
      "\n",
      "N/A\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"META REVIEW:\")\n",
    "print(\"=\" * 50)\n",
    "print(review_data['meta_review'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the Best Paper\n",
    "\n",
    "Now, let's find the highest-rated paper from our reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "BEST PAPER SELECTED:\n",
      "Title: Scientific Peer Reviewer Agents}\n",
      "\n",
      "...\n",
      "Average Rating: 4.00/10\n",
      "Decision: Reject\n",
      "==================================================\n",
      "\n",
      "KEY STRENGTHS OF THE BEST PAPER:\n",
      "\n",
      "Strength 1:\n",
      "The paper proposes a method to train a peer-reviewer agent using a combination of human feedback and LLM-generated reviews.\n",
      "\n",
      "\n",
      "\n",
      "Strength 2:\n",
      "The paper introduces a novel application of LLMs to the iterative and social processes fundamental to the scientific method, and presents a fine-tuning approach to align LLMs with human judgment. The paper also presents a new synthetic dataset (Syn-RS) of high-quality reviews of scientific papers, w...\n",
      "\n",
      "Strength 3:\n",
      "The paper introduces a novel application of LLMs to the iterative and social processes fundamental to the scientific method, and presents a fine-tuning approach to align LLMs with human judgment. The paper also presents a new synthetic dataset (Syn-RS) of high-quality reviews of scientific papers, w...\n",
      "\n",
      "Strength 4:\n",
      "The paper introduces a novel application of LLMs to the iterative and social processes fundamental to the scientific method, and presents a fine-tuning approach to align LLMs with human judgment. The paper also presents a new synthetic dataset (Syn-RS) of high-quality reviews of scientific papers, w...\n"
     ]
    }
   ],
   "source": [
    "# Find the best paper\n",
    "rating_max = 0\n",
    "best_paper_num = 0\n",
    "for i in range(len(review_result)):\n",
    "    if review_result[i] != None:\n",
    "        ratings = review_data['avg_rating']\n",
    "        if ratings > rating_max:\n",
    "            best_paper_num=i\n",
    "            rating_max = ratings\n",
    "\n",
    "\n",
    "print(\"=\" * 50)\n",
    "print(\"BEST PAPER SELECTED:\")\n",
    "print(f\"Title: {papers[best_paper_num]['title']}...\")\n",
    "print(f\"Average Rating: {review_result[best_paper_num]['avg_rating']:.2f}/10\")\n",
    "print(f\"Decision: {review_result[best_paper_num]['paper_decision']}\")\n",
    "print(\"=\" * 50)\n",
    "\n",
    "# Print key strengths from the highest-rated paper's review\n",
    "best_review = review_result[best_paper_num]\n",
    "print(\"\\nKEY STRENGTHS OF THE BEST PAPER:\")\n",
    "for i, strength in enumerate(best_review.get('strength', [])):\n",
    "    if strength:\n",
    "        print(f\"\\nStrength {i+1}:\")\n",
    "        print(strength[:300] + \"...\" if len(strength) > 300 else strength)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this tutorial, we've explored how to use CycleReviewer to evaluate academic research papers. We've seen how the model can:\n",
    "\n",
    "1. Generate detailed reviews that assess papers on multiple dimensions\n",
    "2. Provide specific feedback on strengths and weaknesses\n",
    "3. Assign numerical ratings to quantify paper quality\n",
    "4. Generate meta-reviews that synthesize multiple reviewer perspectives\n",
    "5. Make accept/reject recommendations\n",
    "\n",
    "CycleReviewer represents a significant advancement in automating the peer review process, offering researchers, publishers, and educators a powerful tool for evaluating scientific work. While it can provide valuable feedback and insights, it's important to remember that these automated reviews are best used in conjunction with human expert evaluation, especially for final publication decisions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
